notebook.community

Edit and run 





In [1]:



    


import tensorflow as tf
from tensorflow import data
import shutil
import math
from datetime import datetime
from tensorflow.python.feature_column import feature_column

from tensorflow.contrib.learn import learn_runner
from tensorflow.contrib.learn import make_export_strategy

print(tf.__version__)



    


















    






/Users/khalidsalama/anaconda/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)










    






1.4.0

Steps to use the TF Experiment APIs

  1. Define dataset metadata
  2. Define data input function to read the data from csv files + feature processing
  3. Create TF feature columns based on metadata + extended feature columns
  4. Define an model function with the required feature columns, EstimatorSpecs, & parameters
  5. Run an Experiment with learn_runner to train, evaluate, and export the model
  6. Evaluate the model using test data
  7. Perform predictions & serving the exported model




In [2]:



    


MODEL_NAME = 'reg-model-04'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True

1. Define Dataset Metadata

  • CSV file header and defaults
  • Numeric and categorical feature names
  • Target feature name
  • Unused columns




In [3]:



    


HEADER = ['key','x','y','alpha','beta','target']
HEADER_DEFAULTS = [[0], [0.0], [0.0], ['NA'], ['NA'], [0.0]]

NUMERIC_FEATURE_NAMES = ['x', 'y']  

CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'alpha':['ax01', 'ax02'], 'beta':['bx01', 'bx02']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'target'

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {}".format(TARGET_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))



    


















    






Header: ['key', 'x', 'y', 'alpha', 'beta', 'target']
Numeric Features: ['x', 'y']
Categorical Features: ['alpha', 'beta']
Target: target
Unused Features: ['key']

2. Define Data Input Function

  • Input csv files name pattern
  • Use TF Dataset APIs to read and process the data
  • Parse CSV lines to feature tensors
  • Apply feature processing
  • Return (features, target) tensors

a. parsing and preprocessing logic





In [4]:



    


def parse_csv_row(csv_row):
    
    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))
    
    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)
    
    target = features.pop(TARGET_NAME)

    return features, target

def process_features(features):
    
    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])
    features["xy"] = tf.multiply(features['x'], features['y']) # features['x'] * features['y']
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))
    
    return features

b. data pipeline input function





In [5]:



    


def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL, 
                 skip_header_lines=0, 
                 num_epochs=None, 
                 batch_size=200):
    
    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False
    
    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")
    
    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)
    
    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))
    
    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))
    
    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()
    
    features, target = iterator.get_next()
    return features, target



    











In [6]:



    


features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))



    


















    






* data input_fn:
================
Input file(s): 
Batch size: 200
Epoch Count: None
Mode: eval
Shuffle: False
================

Feature read from CSV: ['x', 'y', 'alpha', 'beta', 'x_2', 'y_2', 'xy', 'dist_xy']
Target read from CSV: Tensor("IteratorGetNext:8", shape=(?,), dtype=float32)

3. Define Feature Columns

The input numeric columns are assumed to be normalized (or have the same scale). Otherwise, a normlizer_fn, along with the normlisation params (mean, stdv or min, max) should be passed to tf.feature_column.numeric_column() constructor.





In [7]:



    


def extend_feature_columns(feature_columns):
    
    # crossing, bucketizing, and embedding can be applied here
    
    feature_columns['alpha_X_beta'] = tf.feature_column.crossed_column(
        [feature_columns['alpha'], feature_columns['beta']], 4)
    
    return feature_columns

def get_feature_columns():
    
    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy() 
    
    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}
        
    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)
        
    if EXTEND_FEATURE_COLUMNS:
        feature_columns = extend_feature_columns(feature_columns)
        
    return feature_columns

feature_columns = get_feature_columns()
print("Feature Columns: {}".format(feature_columns))



    


















    






Feature Columns: {'x': _NumericColumn(key='x', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y': _NumericColumn(key='y', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'x_2': _NumericColumn(key='x_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'y_2': _NumericColumn(key='y_2', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'xy': _NumericColumn(key='xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'dist_xy': _NumericColumn(key='dist_xy', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), 'alpha': _VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'beta': _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), 'alpha_X_beta': _CrossedColumn(keys=(_VocabularyListCategoricalColumn(key='alpha', vocabulary_list=('ax01', 'ax02'), dtype=tf.string, default_value=-1, num_oov_buckets=0), _VocabularyListCategoricalColumn(key='beta', vocabulary_list=('bx01', 'bx02'), dtype=tf.string, default_value=-1, num_oov_buckets=0)), hash_bucket_size=4, hash_key=None)}

4. Define Model Function





In [8]:



    


def regression_model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = 1

    feature_columns = list(get_feature_columns().values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )


    # Create the input layers from the features
    input_layer = tf.feature_column.input_layer(features= features, 
                                                feature_columns= dense_columns+indicator_columns)

#     # Create only 1 hidden layer based on the first element of the hidden_units in the params
#     hidden_layer_size = hidden_units[0]
#     hidden_layer = tf.layers.dense(inputs= input_layer, 
#                                    units=hidden_layer_size, 
#                                    activation=tf.nn.relu)

    # Create a fully-connected layer-stack based on the hidden_units in the params
    hidden_layers = tf.contrib.layers.stack(inputs= input_layer,
                                            layer= tf.contrib.layers.fully_connected,
                                            stack_args= hidden_units)

    # Connect the output layer (logits) to the hidden layer (no activation fn)
    logits = tf.layers.dense(inputs=hidden_layers, 
                             units=output_layer_size)

    # Reshape output layer to 1-dim Tensor to return predictions
    output = tf.squeeze(logits)

    # Provide an estimator spec for `ModeKeys.PREDICT`.
    if mode == tf.estimator.ModeKeys.PREDICT:

        predictions = {
            'scores': output
        }

        export_outputs = {
            'predictions': tf.estimator.export.PredictOutput(predictions)
        }

        return tf.estimator.EstimatorSpec(
            mode=mode,
            predictions=predictions,
            export_outputs=export_outputs)

    # Calculate loss using mean squared error
    loss = tf.losses.mean_squared_error(labels, output)

    # Create Optimiser
    optimizer = tf.train.AdamOptimizer()

    # Create training operation
    train_op = optimizer.minimize(
        loss=loss, global_step=tf.train.get_global_step())

    # Calculate root mean squared error as additional eval metric
    eval_metric_ops = {
        "rmse": tf.metrics.root_mean_squared_error(
            labels, output)
    }

    # Provide an estimator spec for `ModeKeys.EVAL` and `ModeKeys.TRAIN` modes.
    estimator_spec = tf.estimator.EstimatorSpec(mode=mode,
                                                loss=loss,
                                                train_op=train_op,
                                                eval_metric_ops=eval_metric_ops)
    return estimator_spec


def create_estimator(run_config, hparams):
    estimator = tf.estimator.Estimator(model_fn=regression_model_fn, 
                                  params=hparams, 
                                  config=run_config)
    
    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator

5. Run Experiment

a. Define Experiment Function





In [9]:



    


def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn

b. Set HParam and RunConfig





In [10]:



    


TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
NUM_EVAL = 10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[8, 4], 
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS, # evaluate after 100 epochs, i.e., total of 10 evaluations (+ the first one)
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL) 
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")



    


















    






[('batch_size', 500), ('dropout_prob', 0.0), ('hidden_units', [8, 4]), ('num_epochs', 1000)]
Model Directory: trained_models/reg-model-04

Dataset Size: 12000
Batch Size: 500
Steps per Epoch: 24.0
Total Steps: 24000.0
Required Evaluation Steps: 10
That is 1 evaluation step after each 100.0  epochs
Save Checkpoint After 2400 steps

c. Define Serving Function





In [11]:



    


def csv_serving_input_fn():
    
    SERVING_HEADER = ['x','y','alpha','beta']
    SERVING_HEADER_DEFAULTS = [[0.0], [0.0], ['NA'], ['NA']]

    rows_string_tensor = tf.placeholder(dtype=tf.string,
                                         shape=[None],
                                         name='csv_rows')
    
    receiver_tensor = {'csv_rows': rows_string_tensor}

    row_columns = tf.expand_dims(rows_string_tensor, -1)
    columns = tf.decode_csv(row_columns, record_defaults=SERVING_HEADER_DEFAULTS)
    features = dict(zip(SERVING_HEADER, columns))

    return tf.estimator.export.ServingInputReceiver(
        process_features(features), receiver_tensor)

d. Run Experiment via learn_runner





In [12]:



    


if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...") 


tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow() 
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................") 

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[make_export_strategy(
            csv_serving_input_fn,
            exports_to_keep=1
        )]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow() 
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))



    


















    






Removing previous artifacts...
Experiment started at 17:17:57
.......................................
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x121a74b38>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/reg-model-04'}

Estimator Type: <class 'tensorflow.python.estimator.estimator.Estimator'>

WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /Users/khalidsalama/anaconda/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.

* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 500
Epoch Count: 1000
Mode: train
Shuffle: True
================

INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into trained_models/reg-model-04/model.ckpt.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:18:04
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-1
INFO:tensorflow:Finished evaluation at 2017-11-15-17:18:04
INFO:tensorflow:Saving dict for global step 1: global_step = 1, loss = 324.174, rmse = 18.0048
INFO:tensorflow:Validation (step 1): loss = 324.174, rmse = 18.0048, global_step = 1
INFO:tensorflow:loss = 305.581, step = 1
INFO:tensorflow:global_step/sec: 27.7021
INFO:tensorflow:loss = 368.79, step = 101 (0.761 sec)
INFO:tensorflow:global_step/sec: 148.981
INFO:tensorflow:loss = 269.747, step = 201 (0.671 sec)
INFO:tensorflow:global_step/sec: 131.481
INFO:tensorflow:loss = 322.637, step = 301 (0.761 sec)
INFO:tensorflow:global_step/sec: 125.216
INFO:tensorflow:loss = 254.287, step = 401 (0.798 sec)
INFO:tensorflow:global_step/sec: 120.693
INFO:tensorflow:loss = 232.263, step = 501 (0.828 sec)
INFO:tensorflow:global_step/sec: 133.346
INFO:tensorflow:loss = 210.963, step = 601 (0.752 sec)
INFO:tensorflow:global_step/sec: 105.211
INFO:tensorflow:loss = 217.493, step = 701 (0.950 sec)
INFO:tensorflow:global_step/sec: 118.339
INFO:tensorflow:loss = 161.082, step = 801 (0.845 sec)
INFO:tensorflow:global_step/sec: 96.0488
INFO:tensorflow:loss = 158.592, step = 901 (1.041 sec)
INFO:tensorflow:global_step/sec: 107.686
INFO:tensorflow:loss = 142.485, step = 1001 (0.928 sec)
INFO:tensorflow:global_step/sec: 118.178
INFO:tensorflow:loss = 141.336, step = 1101 (0.847 sec)
INFO:tensorflow:global_step/sec: 105.205
INFO:tensorflow:loss = 138.915, step = 1201 (0.951 sec)
INFO:tensorflow:global_step/sec: 127.681
INFO:tensorflow:loss = 157.873, step = 1301 (0.783 sec)
INFO:tensorflow:global_step/sec: 112.202
INFO:tensorflow:loss = 112.564, step = 1401 (0.890 sec)
INFO:tensorflow:global_step/sec: 114.96
INFO:tensorflow:loss = 125.055, step = 1501 (0.869 sec)
INFO:tensorflow:global_step/sec: 135.279
INFO:tensorflow:loss = 101.862, step = 1601 (0.739 sec)
INFO:tensorflow:global_step/sec: 132.51
INFO:tensorflow:loss = 126.73, step = 1701 (0.756 sec)
INFO:tensorflow:global_step/sec: 125.851
INFO:tensorflow:loss = 102.2, step = 1801 (0.794 sec)
INFO:tensorflow:global_step/sec: 129.619
INFO:tensorflow:loss = 119.56, step = 1901 (0.772 sec)
INFO:tensorflow:global_step/sec: 110.474
INFO:tensorflow:loss = 110.547, step = 2001 (0.905 sec)
INFO:tensorflow:global_step/sec: 116.811
INFO:tensorflow:loss = 94.6563, step = 2101 (0.856 sec)
INFO:tensorflow:global_step/sec: 129.81
INFO:tensorflow:loss = 111.647, step = 2201 (0.770 sec)
INFO:tensorflow:global_step/sec: 122.031
INFO:tensorflow:loss = 103.735, step = 2301 (0.819 sec)
INFO:tensorflow:Saving checkpoints for 2401 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 62.227

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:18:27
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-2401
INFO:tensorflow:Finished evaluation at 2017-11-15-17:18:28
INFO:tensorflow:Saving dict for global step 2401: global_step = 2401, loss = 111.019, rmse = 10.5366
INFO:tensorflow:Validation (step 2401): loss = 111.019, rmse = 10.5366, global_step = 2401
INFO:tensorflow:loss = 109.904, step = 2401 (3.406 sec)
INFO:tensorflow:global_step/sec: 38.4282
INFO:tensorflow:loss = 90.4845, step = 2501 (0.804 sec)
INFO:tensorflow:global_step/sec: 126.078
INFO:tensorflow:loss = 93.4027, step = 2601 (0.794 sec)
INFO:tensorflow:global_step/sec: 124.458
INFO:tensorflow:loss = 96.8483, step = 2701 (0.804 sec)
INFO:tensorflow:global_step/sec: 138.629
INFO:tensorflow:loss = 92.6891, step = 2801 (0.721 sec)
INFO:tensorflow:global_step/sec: 141.396
INFO:tensorflow:loss = 93.79, step = 2901 (0.707 sec)
INFO:tensorflow:global_step/sec: 143.568
INFO:tensorflow:loss = 98.7764, step = 3001 (0.696 sec)
INFO:tensorflow:global_step/sec: 121.522
INFO:tensorflow:loss = 107.321, step = 3101 (0.823 sec)
INFO:tensorflow:global_step/sec: 101.757
INFO:tensorflow:loss = 106.702, step = 3201 (0.986 sec)
INFO:tensorflow:global_step/sec: 84.4662
INFO:tensorflow:loss = 90.4981, step = 3301 (1.182 sec)
INFO:tensorflow:global_step/sec: 86.2167
INFO:tensorflow:loss = 102.711, step = 3401 (1.160 sec)
INFO:tensorflow:global_step/sec: 82.4557
INFO:tensorflow:loss = 73.9731, step = 3501 (1.213 sec)
INFO:tensorflow:global_step/sec: 104.218
INFO:tensorflow:loss = 100.379, step = 3601 (0.959 sec)
INFO:tensorflow:global_step/sec: 125.433
INFO:tensorflow:loss = 96.7336, step = 3701 (0.797 sec)
INFO:tensorflow:global_step/sec: 137.894
INFO:tensorflow:loss = 95.0096, step = 3801 (0.725 sec)
INFO:tensorflow:global_step/sec: 135.539
INFO:tensorflow:loss = 90.6274, step = 3901 (0.738 sec)
INFO:tensorflow:global_step/sec: 126.798
INFO:tensorflow:loss = 104.553, step = 4001 (0.790 sec)
INFO:tensorflow:global_step/sec: 113.883
INFO:tensorflow:loss = 95.309, step = 4101 (0.879 sec)
INFO:tensorflow:global_step/sec: 120.287
INFO:tensorflow:loss = 93.0883, step = 4201 (0.830 sec)
INFO:tensorflow:global_step/sec: 136.161
INFO:tensorflow:loss = 82.3985, step = 4301 (0.733 sec)
INFO:tensorflow:global_step/sec: 126.43
INFO:tensorflow:loss = 91.7385, step = 4401 (0.792 sec)
INFO:tensorflow:global_step/sec: 118.332
INFO:tensorflow:loss = 95.2653, step = 4501 (0.846 sec)
INFO:tensorflow:global_step/sec: 130.969
INFO:tensorflow:loss = 91.2204, step = 4601 (0.762 sec)
INFO:tensorflow:global_step/sec: 137.648
INFO:tensorflow:loss = 75.8971, step = 4701 (0.726 sec)
INFO:tensorflow:Saving checkpoints for 4801 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 60.1168

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:18:50
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-4801
INFO:tensorflow:Finished evaluation at 2017-11-15-17:18:52
INFO:tensorflow:Saving dict for global step 4801: global_step = 4801, loss = 99.1767, rmse = 9.95875
INFO:tensorflow:Validation (step 4801): loss = 99.1767, rmse = 9.95875, global_step = 4801
INFO:tensorflow:loss = 91.5243, step = 4801 (3.991 sec)
INFO:tensorflow:global_step/sec: 30.6975
INFO:tensorflow:loss = 96.4061, step = 4901 (0.932 sec)
INFO:tensorflow:global_step/sec: 116.533
INFO:tensorflow:loss = 71.5417, step = 5001 (0.859 sec)
INFO:tensorflow:global_step/sec: 113.577
INFO:tensorflow:loss = 115.22, step = 5101 (0.880 sec)
INFO:tensorflow:global_step/sec: 108.132
INFO:tensorflow:loss = 76.1541, step = 5201 (0.924 sec)
INFO:tensorflow:global_step/sec: 127.932
INFO:tensorflow:loss = 101.792, step = 5301 (0.781 sec)
INFO:tensorflow:global_step/sec: 136.327
INFO:tensorflow:loss = 115.638, step = 5401 (0.734 sec)
INFO:tensorflow:global_step/sec: 140.918
INFO:tensorflow:loss = 80.7678, step = 5501 (0.709 sec)
INFO:tensorflow:global_step/sec: 136.876
INFO:tensorflow:loss = 76.5593, step = 5601 (0.731 sec)
INFO:tensorflow:global_step/sec: 139.89
INFO:tensorflow:loss = 91.0885, step = 5701 (0.715 sec)
INFO:tensorflow:global_step/sec: 133.167
INFO:tensorflow:loss = 85.4326, step = 5801 (0.751 sec)
INFO:tensorflow:global_step/sec: 139.961
INFO:tensorflow:loss = 87.558, step = 5901 (0.715 sec)
INFO:tensorflow:global_step/sec: 136.259
INFO:tensorflow:loss = 98.2536, step = 6001 (0.734 sec)
INFO:tensorflow:global_step/sec: 133.309
INFO:tensorflow:loss = 75.2041, step = 6101 (0.749 sec)
INFO:tensorflow:global_step/sec: 140.177
INFO:tensorflow:loss = 86.4089, step = 6201 (0.713 sec)
INFO:tensorflow:global_step/sec: 142.551
INFO:tensorflow:loss = 98.8525, step = 6301 (0.702 sec)
INFO:tensorflow:global_step/sec: 140.433
INFO:tensorflow:loss = 89.3366, step = 6401 (0.712 sec)
INFO:tensorflow:global_step/sec: 136.153
INFO:tensorflow:loss = 85.1617, step = 6501 (0.735 sec)
INFO:tensorflow:global_step/sec: 142.027
INFO:tensorflow:loss = 100.363, step = 6601 (0.704 sec)
INFO:tensorflow:global_step/sec: 134.214
INFO:tensorflow:loss = 80.5897, step = 6701 (0.744 sec)
INFO:tensorflow:global_step/sec: 131.315
INFO:tensorflow:loss = 89.2196, step = 6801 (0.764 sec)
INFO:tensorflow:global_step/sec: 132.721
INFO:tensorflow:loss = 90.5747, step = 6901 (0.752 sec)
INFO:tensorflow:global_step/sec: 132.441
INFO:tensorflow:loss = 83.1952, step = 7001 (0.754 sec)
INFO:tensorflow:global_step/sec: 137.362
INFO:tensorflow:loss = 73.7089, step = 7101 (0.729 sec)
INFO:tensorflow:Saving checkpoints for 7201 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 62.8855

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:19:12
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-7201
INFO:tensorflow:Finished evaluation at 2017-11-15-17:19:13
INFO:tensorflow:Saving dict for global step 7201: global_step = 7201, loss = 96.9113, rmse = 9.84435
INFO:tensorflow:Validation (step 7201): loss = 96.9113, rmse = 9.84435, global_step = 7201
INFO:tensorflow:loss = 102.963, step = 7201 (3.709 sec)
INFO:tensorflow:global_step/sec: 32.7451
INFO:tensorflow:loss = 87.1191, step = 7301 (0.936 sec)
INFO:tensorflow:global_step/sec: 111.742
INFO:tensorflow:loss = 80.0266, step = 7401 (0.895 sec)
INFO:tensorflow:global_step/sec: 134.62
INFO:tensorflow:loss = 62.3085, step = 7501 (0.742 sec)
INFO:tensorflow:global_step/sec: 135.144
INFO:tensorflow:loss = 106.276, step = 7601 (0.740 sec)
INFO:tensorflow:global_step/sec: 135.778
INFO:tensorflow:loss = 86.2817, step = 7701 (0.737 sec)
INFO:tensorflow:global_step/sec: 131.485
INFO:tensorflow:loss = 94.816, step = 7801 (0.761 sec)
INFO:tensorflow:global_step/sec: 128.826
INFO:tensorflow:loss = 102.25, step = 7901 (0.775 sec)
INFO:tensorflow:global_step/sec: 132.348
INFO:tensorflow:loss = 91.009, step = 8001 (0.756 sec)
INFO:tensorflow:global_step/sec: 140.505
INFO:tensorflow:loss = 76.027, step = 8101 (0.711 sec)
INFO:tensorflow:global_step/sec: 97.8052
INFO:tensorflow:loss = 102.481, step = 8201 (1.025 sec)
INFO:tensorflow:global_step/sec: 78.7297
INFO:tensorflow:loss = 90.9343, step = 8301 (1.272 sec)
INFO:tensorflow:global_step/sec: 75.883
INFO:tensorflow:loss = 77.5593, step = 8401 (1.315 sec)
INFO:tensorflow:global_step/sec: 92.9067
INFO:tensorflow:loss = 73.9644, step = 8501 (1.075 sec)
INFO:tensorflow:global_step/sec: 106.569
INFO:tensorflow:loss = 99.2161, step = 8601 (0.940 sec)
INFO:tensorflow:global_step/sec: 92.4485
INFO:tensorflow:loss = 80.0712, step = 8701 (1.081 sec)
INFO:tensorflow:global_step/sec: 121.173
INFO:tensorflow:loss = 93.2751, step = 8801 (0.825 sec)
INFO:tensorflow:global_step/sec: 139.892
INFO:tensorflow:loss = 91.0283, step = 8901 (0.714 sec)
INFO:tensorflow:global_step/sec: 135.068
INFO:tensorflow:loss = 81.5481, step = 9001 (0.740 sec)
INFO:tensorflow:global_step/sec: 141.43
INFO:tensorflow:loss = 86.362, step = 9101 (0.708 sec)
INFO:tensorflow:global_step/sec: 139.548
INFO:tensorflow:loss = 102.046, step = 9201 (0.716 sec)
INFO:tensorflow:global_step/sec: 127.498
INFO:tensorflow:loss = 96.3737, step = 9301 (0.785 sec)
INFO:tensorflow:global_step/sec: 122.757
INFO:tensorflow:loss = 94.3923, step = 9401 (0.814 sec)
INFO:tensorflow:global_step/sec: 126.044
INFO:tensorflow:loss = 96.458, step = 9501 (0.794 sec)
INFO:tensorflow:Saving checkpoints for 9601 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 75.0063

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:19:35
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-9601
INFO:tensorflow:Finished evaluation at 2017-11-15-17:19:36
INFO:tensorflow:Saving dict for global step 9601: global_step = 9601, loss = 95.7524, rmse = 9.78532
INFO:tensorflow:Validation (step 9601): loss = 95.7524, rmse = 9.78532, global_step = 9601
INFO:tensorflow:loss = 106.336, step = 9601 (3.345 sec)
INFO:tensorflow:global_step/sec: 36.6991
INFO:tensorflow:loss = 107.314, step = 9701 (0.713 sec)
INFO:tensorflow:global_step/sec: 137.973
INFO:tensorflow:loss = 71.1633, step = 9801 (0.725 sec)
INFO:tensorflow:global_step/sec: 135.014
INFO:tensorflow:loss = 87.9739, step = 9901 (0.740 sec)
INFO:tensorflow:global_step/sec: 135.278
INFO:tensorflow:loss = 100.31, step = 10001 (0.740 sec)
INFO:tensorflow:global_step/sec: 126.609
INFO:tensorflow:loss = 92.408, step = 10101 (0.790 sec)
INFO:tensorflow:global_step/sec: 136.781
INFO:tensorflow:loss = 97.7729, step = 10201 (0.730 sec)
INFO:tensorflow:global_step/sec: 135.417
INFO:tensorflow:loss = 80.3, step = 10301 (0.739 sec)
INFO:tensorflow:global_step/sec: 125.671
INFO:tensorflow:loss = 83.4821, step = 10401 (0.796 sec)
INFO:tensorflow:global_step/sec: 108.391
INFO:tensorflow:loss = 72.4306, step = 10501 (0.924 sec)
INFO:tensorflow:global_step/sec: 98.8776
INFO:tensorflow:loss = 83.0199, step = 10601 (1.010 sec)
INFO:tensorflow:global_step/sec: 125.343
INFO:tensorflow:loss = 84.3509, step = 10701 (0.798 sec)
INFO:tensorflow:global_step/sec: 131.566
INFO:tensorflow:loss = 108.604, step = 10801 (0.763 sec)
INFO:tensorflow:global_step/sec: 113.84
INFO:tensorflow:loss = 81.5453, step = 10901 (0.875 sec)
INFO:tensorflow:global_step/sec: 107.895
INFO:tensorflow:loss = 85.9325, step = 11001 (0.927 sec)
INFO:tensorflow:global_step/sec: 131.404
INFO:tensorflow:loss = 90.8869, step = 11101 (0.760 sec)
INFO:tensorflow:global_step/sec: 116.025
INFO:tensorflow:loss = 87.7549, step = 11201 (0.862 sec)
INFO:tensorflow:global_step/sec: 129.617
INFO:tensorflow:loss = 90.1264, step = 11301 (0.772 sec)
INFO:tensorflow:global_step/sec: 125.43
INFO:tensorflow:loss = 85.2653, step = 11401 (0.797 sec)
INFO:tensorflow:global_step/sec: 107.464
INFO:tensorflow:loss = 60.0375, step = 11501 (0.931 sec)
INFO:tensorflow:global_step/sec: 119.667
INFO:tensorflow:loss = 101.51, step = 11601 (0.835 sec)
INFO:tensorflow:global_step/sec: 124.788
INFO:tensorflow:loss = 85.8463, step = 11701 (0.802 sec)
INFO:tensorflow:global_step/sec: 106.763
INFO:tensorflow:loss = 84.845, step = 11801 (0.936 sec)
INFO:tensorflow:global_step/sec: 126.928
INFO:tensorflow:loss = 98.2499, step = 11901 (0.787 sec)
INFO:tensorflow:Saving checkpoints for 12001 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 60.5599

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:19:58
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-12001
INFO:tensorflow:Finished evaluation at 2017-11-15-17:19:59
INFO:tensorflow:Saving dict for global step 12001: global_step = 12001, loss = 95.1853, rmse = 9.7563
INFO:tensorflow:Validation (step 12001): loss = 95.1853, rmse = 9.7563, global_step = 12001
INFO:tensorflow:loss = 97.8091, step = 12001 (4.537 sec)
INFO:tensorflow:global_step/sec: 26.0564
INFO:tensorflow:loss = 66.8572, step = 12101 (0.954 sec)
INFO:tensorflow:global_step/sec: 125.648
INFO:tensorflow:loss = 76.6806, step = 12201 (0.795 sec)
INFO:tensorflow:global_step/sec: 136.423
INFO:tensorflow:loss = 85.0077, step = 12301 (0.732 sec)
INFO:tensorflow:global_step/sec: 103.577
INFO:tensorflow:loss = 71.1743, step = 12401 (0.969 sec)
INFO:tensorflow:global_step/sec: 78.3862
INFO:tensorflow:loss = 85.8499, step = 12501 (1.275 sec)
INFO:tensorflow:global_step/sec: 83.3678
INFO:tensorflow:loss = 97.3313, step = 12601 (1.198 sec)
INFO:tensorflow:global_step/sec: 80.2644
INFO:tensorflow:loss = 74.5801, step = 12701 (1.245 sec)
INFO:tensorflow:global_step/sec: 107.831
INFO:tensorflow:loss = 96.9373, step = 12801 (0.926 sec)
INFO:tensorflow:global_step/sec: 114.27
INFO:tensorflow:loss = 62.2827, step = 12901 (0.875 sec)
INFO:tensorflow:global_step/sec: 137.679
INFO:tensorflow:loss = 86.4258, step = 13001 (0.726 sec)
INFO:tensorflow:global_step/sec: 135.604
INFO:tensorflow:loss = 75.6892, step = 13101 (0.737 sec)
INFO:tensorflow:global_step/sec: 104.43
INFO:tensorflow:loss = 90.7635, step = 13201 (0.959 sec)
INFO:tensorflow:global_step/sec: 78.3317
INFO:tensorflow:loss = 76.7654, step = 13301 (1.276 sec)
INFO:tensorflow:global_step/sec: 106.986
INFO:tensorflow:loss = 73.7493, step = 13401 (0.935 sec)
INFO:tensorflow:global_step/sec: 115.601
INFO:tensorflow:loss = 81.115, step = 13501 (0.864 sec)
INFO:tensorflow:global_step/sec: 134.839
INFO:tensorflow:loss = 93.5551, step = 13601 (0.742 sec)
INFO:tensorflow:global_step/sec: 139.796
INFO:tensorflow:loss = 89.8246, step = 13701 (0.714 sec)
INFO:tensorflow:global_step/sec: 135.746
INFO:tensorflow:loss = 110.005, step = 13801 (0.739 sec)
INFO:tensorflow:global_step/sec: 110.052
INFO:tensorflow:loss = 86.0695, step = 13901 (0.906 sec)
INFO:tensorflow:global_step/sec: 117.753
INFO:tensorflow:loss = 88.913, step = 14001 (0.850 sec)
INFO:tensorflow:global_step/sec: 125.329
INFO:tensorflow:loss = 63.7754, step = 14101 (0.797 sec)
INFO:tensorflow:global_step/sec: 120.181
INFO:tensorflow:loss = 67.4602, step = 14201 (0.834 sec)
INFO:tensorflow:global_step/sec: 120.763
INFO:tensorflow:loss = 87.6832, step = 14301 (0.826 sec)
INFO:tensorflow:Saving checkpoints for 14401 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 57.4726

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:20:23
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-14401
INFO:tensorflow:Finished evaluation at 2017-11-15-17:20:24
INFO:tensorflow:Saving dict for global step 14401: global_step = 14401, loss = 94.825, rmse = 9.73781
INFO:tensorflow:Validation (step 14401): loss = 94.825, rmse = 9.73781, global_step = 14401
INFO:tensorflow:loss = 115.338, step = 14401 (4.270 sec)
INFO:tensorflow:global_step/sec: 26.9336
INFO:tensorflow:loss = 91.855, step = 14501 (1.186 sec)
INFO:tensorflow:global_step/sec: 107.661
INFO:tensorflow:loss = 70.124, step = 14601 (0.930 sec)
INFO:tensorflow:global_step/sec: 95.6159
INFO:tensorflow:loss = 91.2174, step = 14701 (1.044 sec)
INFO:tensorflow:global_step/sec: 123.472
INFO:tensorflow:loss = 79.1345, step = 14801 (0.810 sec)
INFO:tensorflow:global_step/sec: 117.364
INFO:tensorflow:loss = 79.6455, step = 14901 (0.850 sec)
INFO:tensorflow:global_step/sec: 119.987
INFO:tensorflow:loss = 86.6081, step = 15001 (0.834 sec)
INFO:tensorflow:global_step/sec: 102.141
INFO:tensorflow:loss = 77.605, step = 15101 (0.980 sec)
INFO:tensorflow:global_step/sec: 98.9558
INFO:tensorflow:loss = 83.4992, step = 15201 (1.012 sec)
INFO:tensorflow:global_step/sec: 106.431
INFO:tensorflow:loss = 101.17, step = 15301 (0.937 sec)
INFO:tensorflow:global_step/sec: 102.454
INFO:tensorflow:loss = 60.1519, step = 15401 (0.977 sec)
INFO:tensorflow:global_step/sec: 102.958
INFO:tensorflow:loss = 94.1324, step = 15501 (0.971 sec)
INFO:tensorflow:global_step/sec: 115.696
INFO:tensorflow:loss = 94.769, step = 15601 (0.864 sec)
INFO:tensorflow:global_step/sec: 129.466
INFO:tensorflow:loss = 79.8938, step = 15701 (0.772 sec)
INFO:tensorflow:global_step/sec: 137.666
INFO:tensorflow:loss = 70.9775, step = 15801 (0.727 sec)
INFO:tensorflow:global_step/sec: 132.399
INFO:tensorflow:loss = 84.0039, step = 15901 (0.754 sec)
INFO:tensorflow:global_step/sec: 114.34
INFO:tensorflow:loss = 58.0531, step = 16001 (0.876 sec)
INFO:tensorflow:global_step/sec: 94.2941
INFO:tensorflow:loss = 84.2777, step = 16101 (1.061 sec)
INFO:tensorflow:global_step/sec: 86.65
INFO:tensorflow:loss = 83.4427, step = 16201 (1.152 sec)
INFO:tensorflow:global_step/sec: 114.596
INFO:tensorflow:loss = 79.0373, step = 16301 (0.874 sec)
INFO:tensorflow:global_step/sec: 111.071
INFO:tensorflow:loss = 80.4201, step = 16401 (0.899 sec)
INFO:tensorflow:global_step/sec: 102.022
INFO:tensorflow:loss = 92.0403, step = 16501 (0.982 sec)
INFO:tensorflow:global_step/sec: 113.703
INFO:tensorflow:loss = 90.6051, step = 16601 (0.881 sec)
INFO:tensorflow:global_step/sec: 85.5278
INFO:tensorflow:loss = 90.619, step = 16701 (1.169 sec)
INFO:tensorflow:Saving checkpoints for 16801 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 60.5597

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:20:49
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-16801
INFO:tensorflow:Finished evaluation at 2017-11-15-17:20:50
INFO:tensorflow:Saving dict for global step 16801: global_step = 16801, loss = 94.5301, rmse = 9.72266
INFO:tensorflow:Validation (step 16801): loss = 94.5301, rmse = 9.72266, global_step = 16801
INFO:tensorflow:loss = 86.0765, step = 16801 (3.651 sec)
INFO:tensorflow:global_step/sec: 35.2543
INFO:tensorflow:loss = 108.696, step = 16901 (0.836 sec)
INFO:tensorflow:global_step/sec: 133.901
INFO:tensorflow:loss = 68.3136, step = 17001 (0.745 sec)
INFO:tensorflow:global_step/sec: 133.557
INFO:tensorflow:loss = 91.7749, step = 17101 (0.749 sec)
INFO:tensorflow:global_step/sec: 139.598
INFO:tensorflow:loss = 87.6673, step = 17201 (0.716 sec)
INFO:tensorflow:global_step/sec: 142.745
INFO:tensorflow:loss = 101.521, step = 17301 (0.701 sec)
INFO:tensorflow:global_step/sec: 142.548
INFO:tensorflow:loss = 78.7295, step = 17401 (0.701 sec)
INFO:tensorflow:global_step/sec: 135.531
INFO:tensorflow:loss = 69.3208, step = 17501 (0.739 sec)
INFO:tensorflow:global_step/sec: 135.613
INFO:tensorflow:loss = 78.5277, step = 17601 (0.737 sec)
INFO:tensorflow:global_step/sec: 141.893
INFO:tensorflow:loss = 87.7595, step = 17701 (0.705 sec)
INFO:tensorflow:global_step/sec: 136.375
INFO:tensorflow:loss = 88.1816, step = 17801 (0.734 sec)
INFO:tensorflow:global_step/sec: 140.295
INFO:tensorflow:loss = 96.0901, step = 17901 (0.713 sec)
INFO:tensorflow:global_step/sec: 139.509
INFO:tensorflow:loss = 87.9287, step = 18001 (0.716 sec)
INFO:tensorflow:global_step/sec: 125.593
INFO:tensorflow:loss = 80.1272, step = 18101 (0.797 sec)
INFO:tensorflow:global_step/sec: 123.549
INFO:tensorflow:loss = 91.0642, step = 18201 (0.810 sec)
INFO:tensorflow:global_step/sec: 140.226
INFO:tensorflow:loss = 85.7089, step = 18301 (0.712 sec)
INFO:tensorflow:global_step/sec: 134.052
INFO:tensorflow:loss = 89.2168, step = 18401 (0.747 sec)
INFO:tensorflow:global_step/sec: 118.349
INFO:tensorflow:loss = 88.0649, step = 18501 (0.844 sec)
INFO:tensorflow:global_step/sec: 117.601
INFO:tensorflow:loss = 100.742, step = 18601 (0.850 sec)
INFO:tensorflow:global_step/sec: 137.84
INFO:tensorflow:loss = 84.214, step = 18701 (0.726 sec)
INFO:tensorflow:global_step/sec: 137.517
INFO:tensorflow:loss = 79.1987, step = 18801 (0.727 sec)
INFO:tensorflow:global_step/sec: 98.0977
INFO:tensorflow:loss = 89.9931, step = 18901 (1.020 sec)
INFO:tensorflow:global_step/sec: 79.5646
INFO:tensorflow:loss = 104.02, step = 19001 (1.257 sec)
INFO:tensorflow:global_step/sec: 79.8784
INFO:tensorflow:loss = 90.2135, step = 19101 (1.251 sec)
INFO:tensorflow:Saving checkpoints for 19201 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 36.8563

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:21:13
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-19201
INFO:tensorflow:Finished evaluation at 2017-11-15-17:21:15
INFO:tensorflow:Saving dict for global step 19201: global_step = 19201, loss = 94.2922, rmse = 9.71042
INFO:tensorflow:Validation (step 19201): loss = 94.2922, rmse = 9.71042, global_step = 19201
INFO:tensorflow:loss = 84.0389, step = 19201 (7.008 sec)
INFO:tensorflow:global_step/sec: 18.3376
INFO:tensorflow:loss = 79.4139, step = 19301 (1.159 sec)
INFO:tensorflow:global_step/sec: 76.9558
INFO:tensorflow:loss = 93.9544, step = 19401 (1.300 sec)
INFO:tensorflow:global_step/sec: 62.4861
INFO:tensorflow:loss = 97.5546, step = 19501 (1.600 sec)
INFO:tensorflow:global_step/sec: 72.3594
INFO:tensorflow:loss = 85.9556, step = 19601 (1.381 sec)
INFO:tensorflow:global_step/sec: 97.3398
INFO:tensorflow:loss = 92.5018, step = 19701 (1.027 sec)
INFO:tensorflow:global_step/sec: 97.3898
INFO:tensorflow:loss = 78.5447, step = 19801 (1.027 sec)
INFO:tensorflow:global_step/sec: 101.8
INFO:tensorflow:loss = 78.1952, step = 19901 (0.982 sec)
INFO:tensorflow:global_step/sec: 108.08
INFO:tensorflow:loss = 67.3773, step = 20001 (0.924 sec)
INFO:tensorflow:global_step/sec: 130.588
INFO:tensorflow:loss = 75.6347, step = 20101 (0.766 sec)
INFO:tensorflow:global_step/sec: 123.66
INFO:tensorflow:loss = 90.7395, step = 20201 (0.809 sec)
INFO:tensorflow:global_step/sec: 143.303
INFO:tensorflow:loss = 95.1072, step = 20301 (0.698 sec)
INFO:tensorflow:global_step/sec: 133.006
INFO:tensorflow:loss = 99.8141, step = 20401 (0.751 sec)
INFO:tensorflow:global_step/sec: 141.709
INFO:tensorflow:loss = 61.6359, step = 20501 (0.706 sec)
INFO:tensorflow:global_step/sec: 142.72
INFO:tensorflow:loss = 82.7771, step = 20601 (0.700 sec)
INFO:tensorflow:global_step/sec: 110.2
INFO:tensorflow:loss = 97.8497, step = 20701 (0.907 sec)
INFO:tensorflow:global_step/sec: 132.631
INFO:tensorflow:loss = 88.8605, step = 20801 (0.758 sec)
INFO:tensorflow:global_step/sec: 122.167
INFO:tensorflow:loss = 109.574, step = 20901 (0.815 sec)
INFO:tensorflow:global_step/sec: 128.525
INFO:tensorflow:loss = 89.0331, step = 21001 (0.778 sec)
INFO:tensorflow:global_step/sec: 128.299
INFO:tensorflow:loss = 78.3089, step = 21101 (0.779 sec)
INFO:tensorflow:global_step/sec: 127.963
INFO:tensorflow:loss = 83.1385, step = 21201 (0.782 sec)
INFO:tensorflow:global_step/sec: 136.898
INFO:tensorflow:loss = 88.5258, step = 21301 (0.730 sec)
INFO:tensorflow:global_step/sec: 129.286
INFO:tensorflow:loss = 92.417, step = 21401 (0.774 sec)
INFO:tensorflow:global_step/sec: 137.107
INFO:tensorflow:loss = 85.5742, step = 21501 (0.729 sec)
INFO:tensorflow:Saving checkpoints for 21601 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:global_step/sec: 69.1783

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:21:38
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-21601
INFO:tensorflow:Finished evaluation at 2017-11-15-17:21:39
INFO:tensorflow:Saving dict for global step 21601: global_step = 21601, loss = 94.121, rmse = 9.7016
INFO:tensorflow:Validation (step 21601): loss = 94.121, rmse = 9.7016, global_step = 21601
INFO:tensorflow:loss = 77.6509, step = 21601 (3.291 sec)
INFO:tensorflow:global_step/sec: 37.1814
INFO:tensorflow:loss = 85.5089, step = 21701 (0.845 sec)
INFO:tensorflow:global_step/sec: 106.687
INFO:tensorflow:loss = 93.8376, step = 21801 (0.937 sec)
INFO:tensorflow:global_step/sec: 105.935
INFO:tensorflow:loss = 81.9773, step = 21901 (0.944 sec)
INFO:tensorflow:global_step/sec: 109.704
INFO:tensorflow:loss = 103.487, step = 22001 (0.911 sec)
INFO:tensorflow:global_step/sec: 120.046
INFO:tensorflow:loss = 88.4914, step = 22101 (0.834 sec)
INFO:tensorflow:global_step/sec: 124.048
INFO:tensorflow:loss = 101.088, step = 22201 (0.805 sec)
INFO:tensorflow:global_step/sec: 127.909
INFO:tensorflow:loss = 98.1419, step = 22301 (0.782 sec)
INFO:tensorflow:global_step/sec: 133.753
INFO:tensorflow:loss = 69.4008, step = 22401 (0.748 sec)
INFO:tensorflow:global_step/sec: 137.429
INFO:tensorflow:loss = 81.0104, step = 22501 (0.727 sec)
INFO:tensorflow:global_step/sec: 138.787
INFO:tensorflow:loss = 86.5658, step = 22601 (0.720 sec)
INFO:tensorflow:global_step/sec: 138.005
INFO:tensorflow:loss = 76.74, step = 22701 (0.725 sec)
INFO:tensorflow:global_step/sec: 141.652
INFO:tensorflow:loss = 95.1791, step = 22801 (0.706 sec)
INFO:tensorflow:global_step/sec: 139.396
INFO:tensorflow:loss = 68.3581, step = 22901 (0.717 sec)
INFO:tensorflow:global_step/sec: 139.625
INFO:tensorflow:loss = 91.7207, step = 23001 (0.716 sec)
INFO:tensorflow:global_step/sec: 138.493
INFO:tensorflow:loss = 91.1131, step = 23101 (0.722 sec)
INFO:tensorflow:global_step/sec: 139.138
INFO:tensorflow:loss = 85.8931, step = 23201 (0.720 sec)
INFO:tensorflow:global_step/sec: 138.266
INFO:tensorflow:loss = 71.9149, step = 23301 (0.723 sec)
INFO:tensorflow:global_step/sec: 139.454
INFO:tensorflow:loss = 102.607, step = 23401 (0.716 sec)
INFO:tensorflow:global_step/sec: 140.845
INFO:tensorflow:loss = 101.642, step = 23501 (0.710 sec)
INFO:tensorflow:global_step/sec: 137.288
INFO:tensorflow:loss = 90.2794, step = 23601 (0.730 sec)
INFO:tensorflow:global_step/sec: 143.313
INFO:tensorflow:loss = 86.5073, step = 23701 (0.696 sec)
INFO:tensorflow:global_step/sec: 141.96
INFO:tensorflow:loss = 69.6256, step = 23801 (0.705 sec)
INFO:tensorflow:global_step/sec: 137.985
INFO:tensorflow:loss = 92.4355, step = 23901 (0.725 sec)
INFO:tensorflow:Saving checkpoints for 24000 into trained_models/reg-model-04/model.ckpt.
INFO:tensorflow:Loss for final step: 82.5325.

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:21:59
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000
INFO:tensorflow:Finished evaluation at 2017-11-15-17:22:00
INFO:tensorflow:Saving dict for global step 24000: global_step = 24000, loss = 93.9864, rmse = 9.69466
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: b"trained_models/reg-model-04/export/Servo/temp-b'1510766521'/saved_model.pb"
.......................................
Experiment finished at 17:22:01

Experiment elapsed time: 244.561297 seconds

6. Evaluate the Model





In [13]:



    


TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000

train_input_fn = lambda: csv_input_fn(files_name_pattern= TRAIN_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn, steps=1)
train_rmse = str(train_results["rmse"])
print()
print("######################################################################################")
print("# Train RMSE: {} - {}".format(train_rmse, train_results))
print("######################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
valid_rmse = str(valid_results["rmse"])
print()
print("######################################################################################")
print("# Valid RMSE: {} - {}".format(valid_rmse,valid_results))
print("######################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
test_rmse = str(test_results["rmse"])
print()
print("######################################################################################")
print("# Test RMSE: {} - {}".format(test_rmse, test_results))
print("######################################################################################")



    


















    






INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x121a74b38>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/reg-model-04'}

Estimator Type: <class 'tensorflow.python.estimator.estimator.Estimator'>


* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 12000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:22:02
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-15-17:22:03
INFO:tensorflow:Saving dict for global step 24000: global_step = 24000, loss = 85.9296, rmse = 9.26982

######################################################################################
# Train RMSE: 9.26982 - {'loss': 85.929581, 'rmse': 9.2698212, 'global_step': 24000}
######################################################################################

* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 3000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:22:04
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-15-17:22:05
INFO:tensorflow:Saving dict for global step 24000: global_step = 24000, loss = 93.9863, rmse = 9.69466

######################################################################################
# Valid RMSE: 9.69466 - {'loss': 93.986343, 'rmse': 9.6946554, 'global_step': 24000}
######################################################################################

* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5000
Epoch Count: None
Mode: eval
Shuffle: False
================

INFO:tensorflow:Starting evaluation at 2017-11-15-17:22:06
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000
INFO:tensorflow:Evaluation [1/1]
INFO:tensorflow:Finished evaluation at 2017-11-15-17:22:08
INFO:tensorflow:Saving dict for global step 24000: global_step = 24000, loss = 94.3793, rmse = 9.7149

######################################################################################
# Test RMSE: 9.7149 - {'loss': 94.379349, 'rmse': 9.7149038, 'global_step': 24000}
######################################################################################

7. Prediction





In [14]:



    


import itertools

predict_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN, 
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

# predictions = estimator.predict(input_fn=predict_input_fn)
# print("")
# print(list(itertools.islice(predictions, 5)))

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["scores"],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))



    


















    






* data input_fn:
================
Input file(s): data/test-*.csv
Batch size: 5
Epoch Count: None
Mode: infer
Shuffle: False
================

WARNING:tensorflow:Input graph does not contain a QueueRunner. That means predict yields forever. This is probably a mistake.
INFO:tensorflow:Restoring parameters from trained_models/reg-model-04/model.ckpt-24000

Predicted Values: [54.036522, -2.1393905, 19.330648, 2.8917863, 0.18070206]

Serving via the Saved Model





In [16]:



    


import os

export_dir = model_dir +"/export/Servo/"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1] 

print(saved_model_dir)

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predictions"
)

output = predictor_fn({'csv_rows': ["0.5,1,ax01,bx02", "-0.5,-1,ax02,bx02"]})
print(output)



    


















    






trained_models/reg-model-04/export/Servo//1510766521
INFO:tensorflow:Restoring parameters from b'trained_models/reg-model-04/export/Servo//1510766521/variables/variables'
{'scores': array([ 65.32520294,   0.18070206], dtype=float32)}

What can we improve?

  • Use .tfrecords files instead of CSV - TFRecord files are optimised for tensorflow.
  • Use tf.estimator.train_and_evaluate - instead of the experiment object
  • Early Stopping - to avoid overfitting the model to the training set




In [ ]:

Content source: GoogleCloudPlatform/tf-estimator-tutorials

Similar notebooks:

notebook.community | gallery | about